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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  DEFENSE 

(HEALTH  AFFAIRS) 


SUBJECT:  Report  on  the  Acquisition  of  the  Armed  Forces  Health  Longitudinal 
Technology  Application  (Report  No.  D-2006-089) 


We  are  providing  this  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
The  Assistant  Secretary  of  Defense  (Health  Affairs)  comments  were  partially  responsive 
in  that  they  did  not  provide  a  date  for  completion  of  planned  actions.  Therefore,  we 
request  additional  comments  on  all  recommendations  to  include  an  estimated  date  of 
completion  by  June  15,  2006. 

If  possible,  please  send  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only)  to  AudATM@dodig.mil.  Copies  of  the  management  comments  must 
contain  the  actual  signature  of  the  authorizing  official.  We  cannot  accept  the  /  Signed  / 
symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send  classified  comments 
electronically,  they  must  be  sent  over  the  SECRET  Internet  Protocol  Router  Network 
(SIPRNET). 

We  appreciate  the  courtesies  extended  to  the  staff.  Questions  should  be  directed 
to  Ms.  Jacqueline  L.  Wicecarver  at  (703)  604-9077  (DSN  664-9077)  or  Mr.  Sean  A. 

Davis  at  (703)  604-9049  (DSN  664-9049).  The  team  members  are  listed  inside  the  back 
cover.  See  Appendix  E  for  the  report  distribution. 

By  direction  of  the  Deputy  Inspector  General  for  Auditing: 


Assistant  Inspector  General 
Acquisition  and  Contract  Management 


Department  of  Defense  Office  of  Inspector  General 


Report  No.  D-2006-089  May  18,  2006 

(Project  No.  D2005-D000AS-01 17.000) 

Acquisition  of  the  Armed  Forces  Health 
Longitudinal  Technology  Application 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Healthcare  providers;  warfighters;  Armed 
Forces  Health  Longitudinal  Technology  Application  program  officials;  and  individuals 
involved  in  the  requirements  development,  testing,  and  oversight  of  the  Anned  Forces 
Health  Longitudinal  Technology  Application  should  read  this  report.  This  report 
discusses  the  proper  identification  of  the  risks  associated  with  the  integration  of 
commercial  off-the-shelf  software,  as  well  as  the  program  manager’s  emphasis  on  the  use 
of  risk  management,  lessons  learned,  and  perfonnance  monitoring  programs  for  the 
Anned  Forces  Health  Longitudinal  Technology  Application  program. 

Background.  On  November  21,  2005,  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  changed  the  name  of  the  Composite  Health  Care  System  II  to  the  Armed  Forces 
Health  Longitudinal  Technology  Application.  The  Anned  Forces  Health  Longitudinal 
Technology  Application  is  a  medical  and  dental  clinical  information  system  that  will 
generate  and  maintain  a  comprehensive,  lifelong,  computer-based  patient  record  for 
every  soldier,  sailor,  airman,  and  marine;  their  family  members;  and  others  entitled  to 
DoD  military  health  care.  The  Armed  Forces  Health  Longitudinal  Technology 
Application  program  is  expected  to  support  9.2  million  beneficiaries.  As  of  September 
30,  2005,  there  were  7.01  million  patients  with  records  on-line  at  5 1  Medical  Treatment 
Facilities.  The  initial  program  provides  support  capabilities  in  the  outpatient  arena. 
Currently,  the  Armed  Forces  Health  Longitudinal  Technology  Application  program 
management  office  is  planning  for  the  development  of  capabilities  for  inpatient  care.  The 
estimated  cost  of  the  entire  program  is  just  over  $5  billion. 

Results.  Although  the  Armed  Forces  Health  Longitudinal  Technology  Application 
program  management  office  is  using  risk  mitigation  techniques  such  as  risk  management, 
lessons  learned,  and  perfonnance  monitoring,  the  program  remains  at  high  risk  because 
of  the  complexities  of  integrating  commercial  off-the-shelf  software  into  the  existing 
Anned  Forces  Health  Longitudinal  Technology  Application  program.  At  the  time  of  our 
initial  review  in  September  2005,  the  program  management  office  had  not  identified  any 
mitigation  strategies  to  reduce  and  control  risk.  Additionally,  current  strategies  are  not 
sufficient  to  mitigate  the  commercial  off-the-shelf  risk.  As  a  result,  the  Armed  Forces 
Health  Longitudinal  Technology  Application  program  is  vulnerable  to  continued 
increases  in  cost,  extended  schedules  for  implementation,  and  unrealized  goals  in 
performance  from  underestimating  the  difficulties  of  integrating  commercial  off-the-shelf 
products.  See  the  Finding  section  of  the  report  for  detailed  recommendations.  The 
management  controls  that  we  reviewed  were  effective  in  that  we  did  not  identity  any 
material  management  control  weakness. 


Management  Comments  and  Audit  Response.  The  Assistant  Secretary  of  Defense 
(Health  Affairs)  concurred  with  the  draft  recommendations  to  provide  documentation  to 
support  assigned  risks,  provide  justification  and  an  implementation  plan  for  the  high  risk 
assigned  to  Block  III,  and  to  develop  additional  and  more  robust  mitigation  strategies 
associated  with  commercial  off-the-shelf  products.  Although  partially  responsive,  the 
comments  did  not  provide  estimated  completion  dates  for  the  planned  actions. 

We  request  that  the  Assistant  Secretary  of  Defense  (Health  Affairs)  provide  comments  on 
the  final  report  by  June  15,  2006.  A  discussion  of  the  management  comments  is  in  the 
Audit  Results  section  of  the  report,  and  the  complete  text  is  in  the  Management 
Comments  section. 
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Background 


On  November  21,  2005,  the  Assistant  Secretary  of  Defense  (Health  Affairs) 
changed  the  name  of  the  Composite  Health  Care  System  II  (CHCS  II)  to  the 
Anned  Forces  Health  Longitudinal  Technology  Application  (AHLTA).  AHLTA 
is  a  medical  and  dental  clinical  information  system.  The  system  will  generate  and 
maintain  a  comprehensive,  lifelong,  computer-based  patient  record  for  every 
soldier,  sailor,  ainnan,  and  marine;  their  family  members;  and  others  entitled  to 
DoD  military  health  care.  The  computer-based  patient  record  will  provide  real¬ 
time  access  to  individual  and  population  health  care  infonnation  for  health  care 
providers  to  make  informed,  definitive  decisions  on  the  health  care  of  members  of 
the  Armed  Forces  assigned  worldwide,  as  well  as  those  members  deployed  as  part 
of  contingency  operations  at  home  and  abroad.  The  system  will  provide  the 
capability  to  document  patient  medical  care  and  exposure  to  different 
environmental  or  occupational  hazards,  and  to  retrieve  lifelong  medical  records, 
dental  care,  and  immunization  status.  These  electronic  records  will  allow  for 
patient  illness  trend  surveillance,  which  will  help  detect  and  prevent  illness. 

System  Description.  AHLTA  is  expected  to  support  9.2  million  beneficiaries 
with  more  than  132,500  military  and  civilian  medical  personnel  providing 
medical  treatment  at  70  inpatient  facilities  and  828  medical  and  dental  clinics.  As 
of  September  30,  2005,  there  were  7.01  million  patients  with  records  on-line  at 
5 1  Medical  Treatment  Facilities.  Currently,  the  AHLTA  Program  Management 
Office  is  planning  for  the  development  of  capabilities  for  inpatient  care.  In  the 
future,  the  system  will  interface  with  the  Department  of  Veterans  Affairs’ 
HealtheVet-VistA  medical  system. 

Acquisition  Strategy.  The  April  2005  Acquisition  Strategy  stated  that  AHLTA 
is  an  Acquisition  Category  IAM,1  automated  information  system.  The  system 
builds  on  capabilities  of  existing  systems,  phasing  in  their  functions  over  time, 
while  adding  new  capabilities  to  meet  mission  requirements.  AHLTA  initially 
provides  support  capabilities  in  the  outpatient  arena,  while  the  mature  system  will 
extend  those  capabilities  into  the  inpatient  arena.  The  ultimate  goal  is  to  integrate 
all  legacy  CHCS  clinical  functions,  as  well  as  the  functions  of  other  clinical 
applications,  into  AHLTA.  In  order  to  conform  to  the  principles  of  evolutionary 
acquisition,  the  system  is  designed  to  accommodate  changes  and  facilitate  the 
integration  of  future  systems  and  technology,  including  the  integration  of 
commercial  off-the-shelf  (COTS)  products. 

Current  Block  Functions.  AHLTA  will  gather,  store,  and  transmit  computerized 
information  about  a  patient’s  lifetime  health  status  and  health  care.  This 
application  enables  the  rapid  access  and  transfer  of  relevant  patient  information 
for  regional  and  remote  treatment  of  injuries  and  illnesses.  AHLTA  will  also 
support  patient  referrals  to,  and  consultations  with,  specialists  within  a  regional 


'An  Acquisition  Category  IAM  is  a  major  automated  information  system  that  is  estimated  to  require 
program  costs  in  any  single  year  in  excess  of  $32  million  (FY  2000  constant  dollars),  total  program  costs 
in  excess  of  $126  million  (FY  2000  constant  dollars),  or  total  life-cycle  costs  in  excess  of  $378  million 
(FY  2000  constant  dollars),  for  which  the  Milestone  Decision  Authority  is  the  Assistant  Secretary 
Defense  (Networks  and  Information  Integration/DoD  Chief  Information  Officer). 
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area  or  at  distant  locations.  When  fully  operational,  the  computer-based  patient 
record  will  provide  a  paperless,  filmless  health  care  record  that  will  be  a 
confidential  and  comprehensive  record  of  care  for  the  full  continuum  of  theater 
and  peacetime  care.  The  computer-based  patient  record  will  also  provide  links  to 
external  knowledge  sources,  interconnect  network  providers,  and  will  provide 
clinical  decision  support  and  rationale  for  care  rendered.  For  the  first  time,  the 
computer-based  patient  record  will  give  health  care  providers  instant  access  to  a 
continuous  and  coherent  chronology  of  the  health  care  history  of  each  of  their 
patients. 

Schedule  Delay,  The  AHLTA  Full  Operational  Capability  Decision  has  been 
delayed  by  4  years  because  of  Block  1  performance  issues  and  the  Block  2  Dental 
Application  having  to  be  redesigned.  In  April  2004,  Block  1  was  not  meeting  the 
6-second  system  performance  requirement  for  patient  data  retrieval  and  response 
to  user  input  that  are  included  in  the  October  2002  operational  requirement 
document.  The  performance  problems  led  to  the  Navy  and  the  Air  Force  stopping 
deployment  of  AHLTA  at  their  facilities  until  these  perfonnance  issues  were 
resolved.  Based  on  an  analysis  prepared  by  the  Clinical  Information  Technology 
Program  Office  (CITPO),2  the  extreme  performance  degradation  during  this  time 
was  associated  with  database  Input/Output  issues  that  were  eventually  corrected 
with  upgrades  to  the  software  and  hardware.  In  addition  to  the  performance  issues 
causing  schedule  delays  to  the  Full  Operational  Capability,  issues  with  the  Dental 
Application  led  to  schedule  delays  in  fielding  Block  2.  The  performance 
problems  in  Block  1  also  caused  a  delay  in  the  acquisition  of  the  COTS  products 
needed  for  Block  3  capability.  Appendix  C  shows  a  comparison  of  the  three 
AHLTA  Acquisition  Program  baselines. 

Life-Cycle  Cost  Increase.  The  estimated  program  cost  for  AHLTA  has 
increased  by  approximately  $1  billion  (from  $4,023  billion  to  $5,019  billion)  due 
to  the  original  life  cycle  being  extended  by  3  years,  from  FY  2018  to  FY  2021. 
The  extension  of  3  years  being  added  to  the  life  cycle  was  caused  by  the  system 
performance  issues  during  Block  1,  which  led  to  a  delay  in  Block  2  Operational 
Test  and  Evaluation  and  delayed  the  Block  2  Milestone  C  decision.  Additionally, 
a  new  Milestone  B  date  was  required  for  Block  3  because  of  the  delay  in 
acquiring  the  commercial  products  needed  for  the  Block  3  capabilities. 


2  The  Clinical  Information  Technology  Program  Office,  an  office  within  the  Office  of  the  Assistant 
Secretary  of  Defense  (Health  Affairs),  manages  AHLTA. 
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Objectives 


The  audit  was  announced  on  January  25,  2005,  with  the  objective  to  review 
AHLTA  budgeting,  accounting,  performance,  and  user  satisfaction.  In 
April  2005,  the  audit  was  re-scoped  to  review  AHLTA  program  requirements, 
Clinger-Cohen  compliance,  and  management  controls.  The  re-scoped  audit 
objective  was  to  evaluate  program  requirements,  the  related  acquisition  strategy, 
and  system  testing  to  detennine  whether  the  system  was  being  implemented  to 
meet  cost,  schedule,  and  perfonnance  requirements.  We  also  evaluated 
management  controls  as  they  relate  to  AHLTA.  See  Appendix  A  for  a  discussion 
of  the  scope  and  methodology  and  for  infonnation  on  prior  audit  coverage  related 
to  the  objectives. 


Managers’  Internal  Control  Program 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26,  1996, 
and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program  Procedures,” 
August  28,  1996,  require  DoD  organizations  to  implement  a  comprehensive 
system  of  management  controls  that  provides  reasonable  assurance  that  programs 
are  operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  We  reviewed 
Management  Control  Program  documentation  as  it  related  to  AHLTA  to 
accomplish  our  objectives.  The  objective  of  our  audit  was  focused  on  system 
requirements,  Clinger-Cohen  compliance,  and  the  Management  Control  Program 
for  AHLTA. 

Adequacy  of  Management  Controls.  We  found  no  weaknesses  in  the 
Management  Control  Program  for  the  documents  we  reviewed. 

Adequacy  of  Management’s  Self-Evaluation.  We  did  not  discuss  the  adequacy 
of  management’s  self-evaluation  because  we  did  not  find  any  management  control 
weaknesses  for  program  requirements  and  Clinger-Cohen  compliance,  which 
covered  the  objectives  of  our  re-scoped  audit.  The  AHLTA  management  controls 
were  included  in  DoD  Inspector  General  (IG)  Report  No.  D2006-003,  “Security 
Controls  Over  Selected  Military  Health  System  Corporate  Databases,”  October  7, 
2005.  That  report  stated,  “.  .  .  TMA  [TRICARE  Management  Activity]  uses  a 
standard  vulnerability  assessment  form  to  evaluate  all  assessable  units  in  the 
program  offices.  That  assessment  fonn  is  used  to  evaluate  a  range  of  assessable 
units  .  .  .  [but]  does  not  provide  detailed  questions  for  each  assessable  unit  and  is 
not  tailored  to  individual  subject  areas  .  .  ..”  The  report  also  stated, 

Expansion  of  the  MCP  [Management  Control  Program]  self- 
assessment  at  the  Navy,  Air  Force,  Army,  and  TRICARE 
Management  Activity  by  incorporating  specific  electronic, 
physical,  and  personnel  controls  would  assist  activities  in 
complying  with  DoD  guidance.  In  addition,  a  comprehensive 
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self  assessment  would  provide  additional  assurance  that  the 
programs  are  operating  as  intended. 

The  report  recommended  that  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  .  .  .  include  tests  for  electronic,  physical,  and  personnel  controls  in  its 
Management  Control  Plans  to  ensure  compliance  with  DoD  Regulation  5200. 2-R, 
“Personnel  Security  Program,”  January  1987,  and  DoD  Instruction  8500.2, 
“Information  Assurance  Implementation,”  February  6,  2003.  TRICARE 
Management  Activity  concurred  with  the  recommendation. 
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Classification  of  Commercial 
Off-The-Shelf  Risk 


Although  the  AHLTA  Program  Management  Office  used  risk  mitigation 
techniques  such  as  risk  management,  lessons  learned,  and  perfonnance 
monitoring,  the  program  remains  at  high  risk  because  of  the  complexities 
of  integrating  COTS  software  into  the  existing  AHLTA  program. 
Additionally,  at  the  time  of  our  initial  review  in  September  2005,  the 
program  management  office  had  not  identified  any  mitigation  strategies  to 
reduce  and  control  program  risk.  Current  mitigation  strategies  are 
inadequate.  As  a  result,  the  AHLTA  program  is  vulnerable  to  continued 
increases  in  cost,  extended  schedules  for  implementation,  and  unrealized 
goals  in  performance  from  underestimating  the  difficulties  of  integrating 
COTS  products. 


Mitigation  Techniques 


The  AHLTA  Program  Manager  uses  risk  management,  lessons  learned,  and 
performance  monitoring  programs  to  mitigate  cost,  schedule,  and  performance 
risks.  A  risk  management  program  is  used  to  identify,  analyze,  mitigate,  and 
control  risks  before  they  become  problems.  Additionally,  the  program 
management  office  uses  lessons  learned  to  identify  best  practices  or  positive 
experiences  from  resolving  past  problems.  Finally,  the  program  management 
office  uses  benchmark  testing  and  end-to-end  performance  measurement  to 
monitor  systems  perfonnance. 

Risk  Management.  The  AHLTA  Program  Manager  uses  a  risk  management 
program  in  order  to  mitigate  performance  issues  and  user  dissatisfaction,  and 
focuses  on  managing  risks  throughout  the  software  acquisition  life  cycle.  The 
AHLTA  risk  management  process  is  defined  in  the  “CITPO  Risk  Management 
Plan,”  September  20,  2004,  which  provides  guidance  on  identifying,  analyzing, 
mitigating,  and  controlling  risks  before  they  become  problems.  The  CITPO  Risk 
Management  Database  documents  CITPO  program  risks. 

Lessons  Learned.  The  AHLTA  Program  Manager  uses  lessons  learned  to 
mitigate  performance  issues  and  user  dissatisfaction.  The  CITPO  identifies 
lessons  learned  as  resolved  problems,  best  practices,  or  positive  experiences.  The 
CITPO  lessons  learned  database  is  the  central  knowledge  repository  for  CITPO 
lessons  learned.  Lessons  are  captured  on  standardized  forms  and  submitted  by 
subject  matter  experts.  In  addition,  Lessons  Learned  Facilitators  and  Directors 
identify  best  practices  and  industry  standards  on  a  regular  basis. 

Performance  Monitoring.  The  AHLTA  Program  Manager  uses  a  performance 
monitoring  program  in  order  to  mitigate  performance  issues  and  user 
dissatisfaction.  The  AHLTA  Program  Management  Office  uses  benchmark 
testing  to  establish  key  lessons  learned,  tools,  and  processes  from  the  initial  test 
cycle  that  can  be  applied  to  future  testing.  It  also  uses  end-to-end  performance 
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measurement  to  detect  perfonnance  threshold  violations,  to  analyze  and  view 
historical  trends,  and  to  isolate  and  remediate  performance  problems. 


Commercial  Off-The-Shelf  Integration 


The  AHLTA  program  remains  at  high  risk  because  of  its  reliance  on  COTS  to 
fully  satisfy  the  requirements  of  the  program.  To  accomplish  the  requirements  for 
Block  1  of  the  AHLTA  system,  the  AHLTA  Program  Management  Office 
selected  and  procured  COTS  products.  These  products  formed  the  core  of  the 
systems  functions  that  will  be  used  in  all  blocks.  The  AHLTA  Program 
Management  Office  did  not  acquire  any  additional  COTS  products  to  fulfill  the 
requirements  of  Block  2.  However,  the  majority  of  AHLTA  functions  resides  in 
Block  3,  which  involves  the  integration  of  COTS  products.  Specifically,  the 
Block  3  Draft  Capability  Development  Document  requires  that  “the  system  shall 
provide  an  order  entry,  results  documentation,  and  results  retrieval  capability  for 
pharmacy,  laboratory,  and  radiology.”  These  capabilities  will  replace  the  legacy 
system  capabilities  through  the  integration  of  COTS  products.  Additionally,  the 
April  2005  Block  3  Acquisition  Strategy  states  that  COTS  products  will  be 
acquired  and  integrated  into  AHLTA  to  fulfill  the  majority  of  the  critical 
requirements  capabilities.  Therefore,  the  operational  effectiveness  of  Block  3, 
and  thus  the  system  as  a  whole,  relies  on  the  successful  integration  of  COTS 
products. 


Risk  Management 


The  AHLTA  risk  management  process  is  a  six-phase  process  in  which  risks  are 
identified,  analyzed,  planned,  tracked,  controlled,  and  documented  and 
communicated.  Management  action  is  determined  based  on  the  priority  value  of 
the  risk.  Risks  are  prioritized  based  on  the  probability  the  risk  will  occur  and  the 
impact  the  risk  will  have  on  program  cost,  schedule,  and  perfonnance  if  the  risk 
does  occur.  The  AHLTA  Program  Management  Office  reassesses  risk  priority 
levels  when  significant  changes  to  a  risk  occurs.  Risk  management  officials 
within  the  program  office  review  open  risks  to  assess  changing  conditions  and 
identify  significant  changes  in  status.  Program  officials  are  provided  with  routine 
risk  status  reports  during  project  and  team  meetings.  The  risk  management 
officials  use  the  risk  status  reports  to  decide  whether  the  risk  mitigation  plan 
needs  to  be  modified,  the  risk  should  be  closed,  a  contingency  plan  should  be 
invoked,  or  tracking  should  continue. 

The  table  shows  the  CITPO  Risk  Evaluation  Matrix,  which  is  used  to  assign  a 
risk’s  priority  value.  The  values  1  through  5  identify  the  level  of  risk  and  thus, 
the  amount  of  management  action  required  to  mitigate  the  risk. 

•  Priority  Value  1  risks  require  immediate  management  action  and 
mitigation  action  within  3  months. 
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•  Priority  Value  2  risks  do  not  require  immediate  action  and  are  tracked 
by  management. 

•  Priority  Value  3  risks  are  watched4  by  management. 

•  Priority  Value  4  risks  require  monitoring  but  problems  are  not 
anticipated. 

•  Priority  Value  5  risks  do  not  require  action  beyond  normal 
management  attention. 


CITPO  Risk  Evaluation  Matrix 


Probability 


HIGH 

MEDIUM 

LOW 

Occurrence  Is 
Assured 

Occurrence  Is 
Possible 

Occurrence  Is 
Unlikely 

HIGH 

HIGH 

HIGH 

MEDIUM 

Significant 

Impact 

1 

2 

3 

MEDIUM 

HIGH 

MEDIUM 

LOW 

Moderate 

Impact 

2 

3 

4 

LOW 

MEDIUM 

LOW 

LOW 

Little  or  No 
Impact 

3 

4 

5 

COTS  Integration  Risks.  The  AHLTA  Program  Management  Office  considers 
the  integration  of  COTS  products  to  be  a  medium  risk.  The  program  office 


3  The  “CITPO  Risk  Management  Plan,”  September  20,  2004,  defines  “tracked”  as  the  fourth  phase  of  the 
CITPO  Risk  Management  Process.  During  this  phase,  risk  data  is  collected  and  compiled  so  that  it  can  be 
analyzed  for  trends. 

4  The  “CITPO  Risk  Management  Plan,”  September  20,  2004,  defines  “watched”  as  “a  mitigation  approach 
where  management  monitors  a  risk  and  its  attributes  for  significant  change.” 
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identified  a  risk5  associated  with  COTS  integration.  The  risk  states  that  there  is  a 
“potential  concern  that  the  complexity  of  the  COTS  integration  may  result  in 
[program]  costs  being  understated.”  The  program  office  assigned  this  risk  at 
Value  3,  which  indicates  the  risk  has  one  of  the  following:  an  unlikely 
probability  of  occurring  and  a  significant  impact  to  cost,  a  possibility  of  occurring 
and  a  moderate  impact  to  cost,  or  an  assured  probability  of  occurring  and  little  or 
no  impact  to  cost.  The  AHLTA  April  2005  Acquisition  Strategy  states  COTS 
products  will  fulfill  the  majority  of  the  critical  requirements  capabilities. 
Therefore,  we  consider  this  risk  to  be  improperly  prioritized  because  AHLTA 
success  and  full  deployment  relies  heavily  on  the  successful  integration  of  COTS. 

Reprioritize  COTS  Integration  Risk.  The  AHLTA  Program  Management 
Office  should  increase  the  priority  value  of  the  COTS  integration  risk  from 
Priority  Value  3  to  Priority  Value  2.  Prior  DoD  IG  audit  reports  as  well  as  DoD 
and  industry  lessons  learned  on  the  use  and  integration  of  COTS  indicate  that 
when  the  integration  of  COTS  is  more  complex  than  planned,  the  impact  to  cost, 
schedule,  and  performance  is  significant. 

Prior  DoD  IG  Audit  Reports.  DoD  IG  Report  No.  D-2002-124, 
“Allegations  to  the  Defense  Hotline  on  the  Management  of  the  Defense  Travel 
System,”  July  1,  2002,  states  that  the  Defense  Travel  System  Project  Management 
Office  underestimated  the  complexity  of  integrating  COTS  products.  The 
Defense  Travel  System  Project  Management  Office  was  required  to  do  extensive 
developmental  work.  As  a  result,  the  system  was  not  deployed  on  schedule  and 
approximately  $7.5  million  was  spent  unnecessarily  in  order  to  accommodate  the 
schedule  delay.  Another  example  of  the  complexity  of  integrating  COTS 
products  and  the  effect  on  cost  is  cited  in  DoD  IG  Report  No.  D-2002-123, 
“Acquisition  and  Clinger-Cohen  Act  Certification  of  the  Defense  Integrated 
Military  Human  Resources  System,”  June  28,  2002.  The  Defense  Integrated 
Military  Human  Resources  System  Program  Manager  expected  the  COTS 
software  would  require  10  to  20  percent  modification.  The  report  states  that  prior 
DoD  experience  with  COTS  products  indicated  that  it  may  be  unreasonable  to 
expect  to  meet  80  to  90  percent  of  the  required  functionality  with  an  “off-the- 
shelf’  application.  As  a  result,  the  Air  Force  and  Navy  were  required  to  perform 
extensive  modifications  to  achieve  the  required  functionality. 

DoD  and  Industry  Lessons  Learned.  According  to  the  Software 
Engineering  Institute’s  study  entitled  “Commercial  Item  Acquisition: 
Considerations  and  Lessons  Learned,”  June  26,  2000,  the  integration  of  COTS  is 
more  challenging  than  developing  a  custom  capability.  Therefore,  increased 
management  oversight  is  fundamental  to  guarantee  the  success  of  the  integration. 
According  to  the  lessons  learned  guidance,  integrating  COTS  requires  extensive 
expertise.  A  program  management  office  must  not  assume  the  commercial 
product  will  be  integrated  into  the  system  with  minimal  effort.  The  assumption 
could  result  in  user  dissatisfaction  and  schedule  and  cost  overruns.  The  guidance 


5  The  risk  associated  with  COTS  integration  is  identified  in  Risk  Management  Report  2005-020,  October  3, 
2005.  The  audit  team  focused  on  Risk  Management  Report  2005-020  because  we  considered  the  other 
risk  management  reports  to  be  properly  prioritized.  A  Risk  Management  Report  is  a  printout  from  the 
CITPO  Risk  Management  Database  that  identifies  the  risk,  the  impact  of  the  risk,  priority  level  of  the 
risk,  general  comments,  responsible  personnel,  and  risk  mitigation  summaries.  See  Appendix  D  for  Risk 
Management  Report  2005-020. 
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also  states  that  an  incomplete  evaluation  of  commercial  items  can  affect  program 
planning  in  unexpected  ways.  Specifically,  vendor  deficiencies  or  new  versions 
of  the  product  can  delay  the  schedule  and  increase  program  costs. 

The  AHLTA  Program  Management  Office  must  also  be  aware  of  the  affect  on 
cost  if  the  commercial  products  become  obsolete  or  require  new  versions  or 
upgrades. 

Mitigation  Strategies.  According  to  Risk  Management  Report  ID  2005-020, 
“COTS  Integration,”  October  3,  2005,  the  Program  Management  Office  did  not 
have  a  mitigation  strategy  associated  with  the  identified  COTS  integration  risk. 
The  CITPO  Risk  Management  Plan  states  risk  information  should  be  translated 
into  decisions  and  both  present  and  future  mitigation  actions.  The  CITPO  Risk 
Management  Plan  states  these  actions  should  then  be  implemented.  Mitigation 
strategies  are  used  to  reduce  risk  by  either  reducing  the  impact  or  the  probability, 
or  both,  of  the  risk.  The  program  management  office  stated  that  it  was  evaluating 
mitigation  strategies.  The  lack  of  a  mitigation  strategy  could  potentially  increase 
program  life-cycle  costs,  schedule,  and  performance. 


AHLTA  Program  Management  Office  Response  to  Discussion 
Draft 


In  response  to  a  discussion  draft  of  this  report,  the  AHLTA  Program  Management 
Office  staff  commented  that  we  were  incorrect  in  stating  that  they  had  not 
developed  any  mitigation  strategies  associated  with  COTS  integration. 
Specifically,  they  responded,  “CITPO  has  identified  COTS  integration  as  a 
medium  level  program  risk  and  developed  corresponding  mitigation  strategies.” 
Additionally,  they  suggested  that  our  recommendation  could  be  changed  to, 
“Develop  additional,  more  robust  mitigation  strategies  to  further  reduce  and 
control  this  risk.”  In  response  to  our  request  for  additional  information  to  support 
their  statement,  the  program  office  staff  provided  an  updated  copy  of  their  COTS 
Integration  Risk  Management  Report,  March  2,  2006,  which  showed  that  on 
September  28,  2005,  the  project  officer  approved  opening  two  mitigation 
strategies  identified  by  the  CHCS  II  Project  Team.  Appendix  D  contains  the 
October  3,  2005,  and  March  2,  2006,  Risk  Management  Reports. 

Supporting  Documentation  for  Mitigation  Strategies.  The  Risk  Management 
Report  is  the  only  documentation  provided  by  AHLTA  to  support  its  position  that 
it  had  developed  two  mitigation  strategies  to  address  the  COTS  integration  risk. 
The  stated  mitigation  strategies,  below,  are  not  sufficient  to  mitigate  the  COTS 
integration  risk: 

•  Mitigation  Strategy  No.  2005-020-1:  Coordinate  across  Information 
Management  and  Information  Technology  teams  during  the  FY  2008 
Program  Objectives  Memorandum  development  cycle;  and 

•  Mitigation  Strategy  No.  2005-020-2:  Address  the  risk  in  the  Program 
Objectives  Memorandum  submissions. 
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Specifically,  the  mitigation  strategy  documents  provided  did  not  include  what 
actions  must  be  taken,  the  level  of  effort  and  materials  required,  the  estimated  cost 
to  implement  the  plan,  a  proposed  schedule  showing  the  proposed  start  date,  the 
time  phasing  of  significant  risk  reduction  activities,  the  completion  date,  and 
relationships  to  significant  activities  and  milestones  as  recommended  by  the 
“DoD  Risk  Management  Guide  for  DoD  Acquisition,”  Fifth  Edition,  version  2.0, 
June  2003. 

Conflicting  Priority  Values  of  the  Risk.  The  October  3,  2005,  Risk 
Management  Report  identifies  COTS  integration  as  Priority  Value  3,  which, 
according  to  the  CITPO  Risk  Evaluation  Matrix  discussed  on  page  7  of  this 
report,  is  a  medium  risk.  The  program  management  office  comments  to  the 
discussion  draft  report,  February  23,  2006,  also  identified  COTS  integration  as  a 
medium  risk.  However,  according  to  the  March  2,  2006,  Risk  Management 
Report,  the  program  management  office  had  raised  the  risk  level  of  COTS 
integration  from  Priority  Value  3  to  Priority  Value  2  during  a  November  1,  2005, 
in-process  review  based  on  the  complexity  of  integrating  COTS  products  in 
Block  3.  Subsequently,  on  January  17,  2006,  the  program  management  office 
raised  the  COTS  integration  risk  from  Priority  Value  2  to  Priority  Value  1  based 
on  the  advice  of  the  CHCS  II  Project  Officer  and  the  CHCS  II  Engineering  Team 
regarding  cost  and  complexity  concerns  with  the  COTS  integration. 

According  to  the  CITPO  Risk  Management  Plan,  Priority  Value  1  risks  indicate 
the  probability  of  occurrence  is  assured  and  the  impact  to  cost,  schedule,  or 
performance  is  severe.  Risks  designated  as  Priority  Value  1  require  an  immediate 
change  in  current  project  activities  in  order  to  reduce  or  eliminate  the  risk. 
Management  action  is  required  within  3  months  to  begin  implementing 
mitigations.  If  the  COTS  integration  risk  was  increased  from  Priority  Value  2  to 
Priority  Value  1  on  January  17,  2006,  as  stated  in  the  March  2,  2006,  Risk 
Management  Plan,  the  AHLTA  Program  Management  Office  had  until  April  17, 
2006,  to  begin  implementing  mitigations.  The  Program  Management  Office  did 
not  provide  project  activities  to  reduce  or  eliminate  the  COTS  integration  risk  in 
its  April  7,  2006,  response  to  the  draft  report. 

The  program  management  office’s  rationale  for  increasing  the  risk  to  this  level  is 
uncertain.  The  CITPO  definition  of  a  Priority  Value  1  is  that  the  risk  must  be 
assured  to  occur  and  be  of  significant  impact.  However,  the  program 
management  office  did  not  provide  documentation  that  supported  the  occurrence 
of  this  risk  is  assured.  Block  3  of  the  program  had  not  yet  received  a  Milestone  B 
decision  to  enter  System  Development  and  Demonstration,  the  acquisition  phase 
in  which  integration  risk  is  reduced.  Additionally,  during  the  Systems  Integration 
portion  of  the  System  Development  and  Demonstration  phase,  subsystems  are 
integrated,  design  details  are  completed,  and  system-level  risk  is  reduced. 

Finally,  we  believe  that  the  mitigation  strategies  identified  in  the  March  3,  2006, 
Risk  Management  Report  will  not  satisfactorily  mitigate  this  risk  in  the  allotted 
time  frame  because  Program  Objectives  Memorandum  submissions  will  not  be 
delivered  to  the  Comptroller  until  August  2006,  which  is  past  the  April  17,  2006, 
implementation  deadline. 
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Conclusion 


Integration  of  Block  3  COTS  software  remains  a  significant  risk  for  the  successful 
completion  of  AHLTA.  Two  prior  audit  reports  on  other  systems  have  shown 
that  the  impact  to  cost  is  substantial  if  COTS  products  require  extensive 
unplanned  developmental  work.  Additionally,  DoD  and  industry  lessons  learned 
state  that  the  impact  to  DoD  systems  cost,  schedule,  and  performance  is 
significant  if  the  integration  of  COTS  products  is  more  complex  than  planned. 

The  program  management  office  response  to  the  discussion  draft  report  stated  that 
COTS  integration  was  a  medium  risk;  however,  documentation  used  to  support 
that  statement  identified  that  the  risk  was  a  Priority  Value  1  (high).  Additionally, 
the  provided  information  did  not  support  that  elevation.  Without  a  mitigation 
strategy  that  includes  such  information  as  the  required  actions  needed  to  mitigate 
the  risk,  the  level  of  effort  and  materials  required,  the  estimated  cost,  and  the 
proposed  implementation  schedule,  the  risk  of  increased  program  costs  as  a  result 
of  the  unsuccessful  COTS  integration  is  increased  as  the  impact  and  probability  of 
the  risk  is  not  reduced. 


Management  Comments  on  the  Finding  and  Audit  Response 


Management  Comments  on  the  Identification  of  Mitigation  Strategies,  The 

Assistant  Secretary  of  Defense  (Health  Affairs)  disagreed  with  the  statement  that 
the  program  management  office  had  not  identified  any  mitigation  strategies  to 
reduce  and  control  risk.  According  to  the  Assistant  Secretary,  the  program 
management  office  did  identify  several  mitigation  strategies  and  recommended 
that  the  additional  COTS-related  Risk  Management  Reports  be  included  in 
Appendix  D  of  our  final  report.  The  complete  text  of  the  Assistant  Secretary  is  in 
the  Management  Comments  section  of  this  report. 

Audit  Response.  The  audit  team  focused  specifically  on  Risk  Management 
Report  ID  2005-020,  “COTS  Integration,”  October  3,  2005,  because  we 
considered  the  other  risk  management  reports  related  to  integration  of  COTS 
products  to  be  properly  prioritized.  Although  the  CITPO  Program  Management 
Office  was  evaluating  mitigation  strategies  for  this  risk  at  the  time  of  our  initial 
review  in  September  2005,  they  had  not  developed  any. 

In  response  to  the  Discussion  Draft  Report,  CITPO  provided  the  audit  team  with 
an  updated  Risk  Management  Report  ID  2005-020,  March  2,  2006,  as  well  as 
three  additional  COTS-related  risk  reports  referred  to  by  the  Assistant  Secretary 
of  Defense  (Health  Affairs).  The  two  mitigation  strategies  provided  in  the 
updated  Risk  Management  Report  ID  2005-020  were  not  sufficient  to  mitigate  the 
risk.  Specifically,  the  mitigation  strategies  did  not  include  what  actions  must  be 
taken,  the  level  of  effort  and  materials  required,  the  estimated  cost  to  implement 
the  plan,  a  proposed  schedule  showing  the  proposed  start  date,  the  time  phasing  of 
significant  risk  reduction  activities,  the  completion  date,  and  relationships  to 
significant  activities  and  milestones  as  recommended  by  the  “DoD  Risk 
Management  Guide  for  DoD  Acquisition,”  Fifth  Edition,  version  2.0,  June  2003. 
Therefore,  we  recommended  that  the  CITPO  Program  Management  Office 
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develop  more  robust  mitigation  strategies  for  COTS  integration  risk,  Risk 
ID  2002-020. 


The  additional  risk  management  reports  included  in  the  CITPO  response  to  the 
Discussion  Draft  Report  were  not  included  in  the  Draft  Report  because  they  did 
not  relate  to  our  review  of  Risk  Management  Report  ID  2005-020.  We  updated 
the  report  to  clarify  the  COTS  integration  risk  of  our  review  was  Risk 
Management  Report  ID  2005-020.  However,  at  the  request  of  the  program 
management  office,  we  included  the  following  Risk  Management  Reports  in 
Appendix  D:  CHCS  II,  “Block  Ill-Lab  AP/COTS  Interoperability,”  Risk  ID 
2004-080,  January  1,  2005;  CHCS  II,  “COTS  Integration/Convergence,”  Risk  ID 
2004-085,  January  1,  2005;  and  CHCS  II,  “COTS  Integration,”  Risk  ID  2004- 
086,  June  10,  2004. 

Management  Comments  on  Contradiction  of  Risk  Management  Reports. 

The  Assistant  Secretary  of  Defense  (Health  Affairs)  disagreed  with  our  analysis 
of  the  October  3,  2005,  and  March  2,  2006,  Risk  Management  Reports,  stating 
that  he  did  not  see  a  contradiction  in  these  reports.  According  to  the  Assistant 
Secretary,  the  October  3,  2005,  Risk  Management  Report  shows  the  continuing 
evaluation  of  the  mitigations  strategies  which  were  initiated  on  September  28, 

2005.  The  current  Risk  Management  Report,  March  2,  2006,  provides  a  traceable 
timeline  of  the  changes  to  the  Risk  Management  Report  from  when  the  risk  was 
established. 

Audit  Response.  We  accept  the  explanation  the  AHLTA  Program  Management 
Office  provided.  Therefore,  we  have  removed  the  statement  that  the  March  2, 

2006,  Risk  Management  Report  contradicts  the  one  provided  to  the  audit  team  on 
October  3,  2005.  However,  at  the  time  of  our  initial  review  in  September  2005, 
there  were  no  mitigation  strategies  developed  for  the  risk  of  the  integration  of 
COTS.  Also,  the  mitigation  strategies  in  the  March  2,  2006,  Risk  Management 
Report  were  inadequate  in  that  they  did  not  contain  the  recommendations  of  the 
DoD  Risk  Management  Guide  for  DoD  Acquisition.  In  addition,  the  program 
management  office  did  not  provide  the  significant  activities  and  milestones 
recommended  by  the  DoD  Risk  Management  Guide.  Recommendation  3 
requested  that  the  program  office  develop  more  robust  mitigation  strategies  in 
accordance  with  the  CITPO  Risk  Management  Plan. 

Management  Comments  on  Conflicting  Priority  Values  of  the  Risk.  The 

Assistant  Secretary  of  Defense  (Health  Affairs)  stated  that  “Conflicting  Priority 
Values  of  the  Risk”  is  a  misleading  statement  and  suggested  changing  the  word 
conflicting  to  adjusting.  In  addition,  he  stated  that  the  audit  team  had  apparently 
misinterpreted  and  taken  out  of  context  the  statement  that  CITPO  had  identified 
COTS  integration  as  a  medium  risk  and  developed  corresponding  mitigation 
strategies.  The  Assistant  Secretary  stated  the  statement  was  intended  to  refute  the 
information  contained  in  the  draft  report  that  “the  program  office  had  not 
identified  any  mitigation  strategies.”  Also,  he  stated  that  CITPO  provided  four 
COTS  risk  management  reports  with  risk  priorities  ranging  from  medium  to  high 
risk  in  response  to  the  discussion  draft. 

Audit  Response.  We  agree  that  the  risk  level  for  Risk  Management  Report 
ID  2005-020,  “COTS  Integration,”  October  3,  2005,  has  been  adjusted.  However, 
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the  AHLTA  Program  Management  Office’s  written  response  did  not  correlate 
with  the  actual  risk  report  provided.  The  CITPO  response  to  the  discussion  draft 
specifically  states  that  COTS  integration  is  a  medium-level  program  risk. 
Therefore,  we  request  the  Program  Manager,  Armed  Forces  Health  Longitudinal 
Technology  Application  provide  documentation  that  supports  the  movement  of 
COTS  integration  to  higher  risk  levels  as  stated  in  Recommendation  1  of  the  draft 
report.  In  addition,  we  acknowledge  that  CITPO  provided  four  COTS  Risk 
Management  Reports  with  risk  priorities  ranging  from  medium  to  high.  However, 
our  identification  of  conflicting  risk  levels  was  focused  on  Risk  Management 
Report  ID  2005-020,  “COTS  Integration,”  October  3,  2005. 

Recommendations 

We  recommend  that  the  Program  Manager,  Armed  Forces  Health 
Longitudinal  Technology  Application: 

1.  Provide  documentation  that  supports  the  program  management 
office  decisions  on  November  1,  2005,  and  January  17,  2006,  that  increased 
the  risk  priority  value  for  commercial  off-the-shelf  product  integration  into 
the  Armed  Forces  Health  Longitudinal  Technology  Application  from 
Priority  Value  3  (medium)  to  Priority  Value  2  (high),  and  from  Priority 
Value  2  (high)  to  Priority  Value  1  (high). 

2.  Provide  justification  and  an  implementation  plan  for  the  Priority 
Value  1  (high)  risk  assigned  to  Block  3. 

3.  Develop  additional  or  more  robust  mitigation  strategies  that 
address  the  commercial  off-the-shelf  product  integration  Priority  Value  1 
(high)  risk  in  accordance  with  the  CITPO  Risk  Management  Plan.  These 
mitigation  strategies  should,  at  a  minimum,  contain  the  recommendations 
included  in  the  “DoD  Risk  Management  Guide  for  DoD  Acquisition,  Fifth 
Edition,  version  2.0,  June  2003. 

Management  Comments.  The  Assistant  Secretary  of  Defense  (Health  Affairs) 
concurred  stating  that  the  AHLTA  Program  Manager  will  provide  appropriate 
documentation  to  support  the  assignment  of  risk  priorities  associated  with 
commercial  off-the-shelf  product  integration,  provide  justification  and  an 
implementation  plan  for  the  Priority  Value  1  (high)  risk  assigned  to  the  AHLTA 
Block  3,  and  continue  to  develop  additional  and  more  robust  mitigation  strategies 
that  address  the  commercial  off-the-shelf  product  integration  Priority  Value  1 
(high)  risk. 

Audit  Response.  Although  the  Assistant  Secretary  of  Defense  (Health  Affairs) 
concurred  with  the  recommendations,  the  comments  are  partially  responsive  in 
that  a  completion  date  for  the  planned  actions  was  not  provided.  Additionally,  the 
date  for  implementation  of  mitigations  for  COTS  integration  has  passed.  The 
AHLTA  Program  Management  Office  did  not  provide  project  activities  to  reduce 
or  eliminate  the  COTS  integration  risk  in  its  April  7,  2006,  response  to  the  draft 
report.  Therefore,  we  request  that  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  provide  the  completion  date  for  the  planned  actions  to  the  final  report  by 
June  15,  2006. 
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Appendix  A.  Scope  and  Methodology 


We  reviewed  laws,  policies,  guidance,  and  documentation  dated  from  January  24, 
1997  through  March  2,  2006,  related  to  the  system  requirements  of  AHLTA.  To 
accomplish  our  specific  objective,  we  met  with  officials  from  the  AHLTA 
Program  Management  Office,  the  Clinical  Infonnation  Technology  Program 
Office,  the  Joint  Medical  Information  System  Program  Executive  Office,  and 
officials  from  the  Offices  of  the  Assistant  Secretary  of  Defense  (Health  Affairs), 
the  Assistant  Secretary  of  Defense  (Networks  and  Infonnation  Integrations/DoD 
Chief  Information  Officer),  the  Joint  Chiefs  of  Staff,  the  Joint  Interoperability 
Test  Command,  the  Anny  Test  and  Evaluation  Command,  and  the  Director  of 
Operational  Test  and  Evaluation. 

We  began  the  audit  with  an  overall  objective  to  review  budgeting,  accounting, 
performance,  and  user  satisfaction  of  the  AHLTA  to  determine  whether  the 
system  was  being  implemented  to  meet  cost,  schedule,  and  performance 
requirements.  However,  during  the  audit  the  objective  was  re-scoped  to  only 
review  system  requirements,  Clinger-Cohen  compliance,  and  management 
controls.  Specifically,  we  reviewed  operational  requirements,  acquisition 
strategy,  and  operational  and  developmental  testing.  The  re-scoping  of  the  audit 
resulted  from  a  meeting  held  on  April  10,  2005,  between  management  from  the 
Department  of  Defense  Office  of  Inspector  General  and  officials  from  the  Office 
of  the  Assistant  Secretary  of  Defense  (Networks  and  Infonnation  Integration/DoD 
Chief  Information  Officer). 

We  perfonned  this  audit  from  January  2005  through  March  2006  in  accordance 
with  generally  accepted  government  auditing  standards.  We  collected  the 
information  for  the  audit  through  meetings,  e-mails,  and  briefings  with  the 
personnel  stated  above.  We  reviewed  laws,  policies,  guidance,  and 
documentation  for  each  area  we  reviewed  during  the  audit.  Specifically  we 
reviewed: 

•  Public  Law  108-287,  “Department  of  Defense  Appropriations  Act  for 
Fiscal  Year  2005,”  August  5,  2004;  Public  Law  104-106,  “National 
Defense  Authorization  Act  for  Fiscal  Year  1996,”  February  10,  1996; 
Office  of  Management  and  Budget  Circular  A-l  1,  “Preparation  and 
Submission  of  Budget  Estimates,”  July  12,  1999;  Chairman  of  the 
Joint  Chiefs  of  Staff  Instruction  3170.D,  “Joint  Capabilities  Integration 
and  Development  System,”  March  12,  2004;  Federal  Acquisition 
Regulation  Part  39,  “Acquisition  of  Information  Technology,”  March 
2005;  DoD  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition 
System,”  May  12,  2003;  Defense  Acquisition  Guidebook,  April  27, 
2005;  and  the  Acquisition  Community  Connection  Web  site  for 
compliance  with  the  Clinger-Cohen  requirements. 

•  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3 170.0  IB, 
“Requirements  Generation  System,”  April  15,  2001;  Chainnan  of  the 
Joint  Chiefs  of  Staff  Instruction  3 170.0 1C,  “Joint  Capabilities 
Integration  and  Development  System,”  June  24,  2003;  CHCS  II 
Mission  Needs  Statement,  January  28,  1997;  Analysis  of  Alternatives, 
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April  28,  1998;  Operational  Requirement  Documents,  January  24, 
1997;  September  18,  2000;  October  30,  2002;  and  May  4,  2004;  Draft 
Capability  Development  Document  for  Block  3,  January  2005;  and 
Director,  Operational  Test  and  Evaluation  Memorandum  for  Block  2, 
February  24,  2005,  to  detennine  the  validity  of  the  AHLTA 
requirements. 

•  DoD  Directive  8320.2,  “Data  Sharing  in  a  Net-Centric  Department  of 
Defense,”  December  2,  2004;  CHCS  II  Test  and  Evaluation  Master 
Plan  for  Block  2,  September  16,  2004;  CHCS  II  System  Evaluation 
Reports  for  Block  1  and  Block  2,  September  26,  2002,  and 
February  18,  2005;  and  Joint  Interoperability  Test  Certifications  for 
Block  1  and  Block  2,  October  10,  2003,  and  March  28,  2005,  for 
potential  testing  issues  or  problems. 

•  The  CHCS  II  Acquisition  Strategies  for  Block  2  and  Block  3, 
September  16,  2003,  and  April  12,  2005;  Acquisition  Program 
Baselines,  January  27,  2003;  November  17,  2003;  and  May  27,  2005; 
Acquisition  Decision  Memorandums  for  Block  1  and  Block  2, 

January  28,  1997;  February  20,  1998;  December  23,  1998;  January  28, 
2003;  June  13,  2003;  November  17,  2003;  and  May  27,  2005;  and 
CHCS  II  Defense  Acquisition  Executive  Summary  Reports,  Third 
Quarter  FY  2004  through  First  Quarter  FY  2006  for  potential 
schedule,  performance,  and  cost  issues. 

•  The  CITPO  Risk  Management  Plan,  September  20,  2004;  the  CHCS  II 
Risk  Management  Plan,  September  29,  2003;  the  CHCS  II  Risk 
Management  Database,  October  6,  2005;  the  CITPO  Fessons  Fearned 
Database  and  Reports,  September  23,  2005;  the  CHCS  II  Performance 
Monitoring  Program,  September  27,  2005;  the  AHFTA  Active  Risk 
Management  Report,  October  3,  2005;  and  the  AHFTA  Risk 
Management  Report,  March  2,  2006,  for  an  understanding  of  the 
mitigation  techniques  being  emphasized  by  the  AHFTA  Program 
Manager  to  gain  greater  control  over  potential  schedule  delays  and 
increases  in  program  costs. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Use  of  Technical  Assistance.  Two  computer  engineers  from  the  Software 
Engineering  Branch,  Technical  Assessment  Division,  for  Investigative  Policy  and 
Oversight,  DoD  Office  of  Inspector  General,  assisted  in  the  audit.  The  computer 
engineers  assisted  the  audit  team  by  determining  that  software  and  hardware 
problems  encountered  by  AHFTA  during  the  full  deployment  of  Block  1 
functionalities  were  not  COTS-related. 

Government  Accountability  Office  High-Risk  Area.  The  Government 
Accountability  Office  (GAO)  has  identified  several  high-risk  areas  in  DoD.  This 
report  provides  coverage  of  the  DoD  approach  to  the  business  transfonnation 
high-risk  area. 
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Prior  Coverage 


During  the  last  7  years,  GAO  and  the  DoD  IG  have  issued  four  reports  discussing 
the  Composite  Health  Care  System  II. 


GAO 


GAO  Report  No.  GAO-04-69 1R,  “Post-Hearing  Questions  on  VA/DoD  Health 
Data  Exchange,”  May  14,  2004 

GAO  Report  No.  GAO-02-345,  “Greater  Use  of  Best  Practices  Can  Reduce  Risks 
in  Acquiring  Defense  Health  Care  System,”  September  2002 


DoD  IG 

DoD  IG  Report  No.  D-2001-038,  “Allegations  Relating  to  the  Procurement  of  a 
Report  Module  for  the  Composite  Health  Care  System  II,”  January  29,  2001 

DoD  IG  Report  No.  D- 1999-068,  “Acquisition  Management  of  the  Composite 
Health  Care  System  II  Automated  Information  System,”  January  21,  1999 
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Appendix  B.  Armed  Forces  Health  Longitudinal 

Technology  Application  Acquisition 
Strategy 

The  table  below  represents  the  AHLTA  acquisition  strategy.  The  AHLTA 
system’s  acquisition  is  divided  into  three  blocks,  which  are  divided  into  multiple 
releases. 


Current  AHLTA  Functional  Block  Strategy 

Block  1 

Block  2 

Block  3 

Encounter  Documentation 

Release  1 

Release  1 

Order  Entry  and  Results 

Spectacle  Request 

Pharmacy 

Retrieval 

Transmission  System  II 

Release  2 

Encounter  Coding  Support 

Release  2 

Laboratory 

Consult  Tracking 

Dental  Charting  and 

Alerts  and  Reminders 

Automated  Clinical 

Documentation' 

Anatomic  Pathology 

Release  3 

Practice  Guidelines1 

Radiology 

Role-Based  Security 

Release  43 

Elealth  Data  Security 

Elealth  Data  Dictionary 

Inpatient  Charting  and 
Documentation 

Occupational  Health 

Master  Patient  Index 

Ad  Hoc  Query  Ability 

Surveillance 

'Moved  from  Block  2,  Release  2. 

2Moved  from  Block  2,  Release  1. 

3Result  of  Block  4  being  merged  into  Block  3. 
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Appendix  C.  Armed  Forces  Health  Longitudinal 

Technology  Application  Acquisition 
Program  Baselines 


The  chart  below  shows  the  AHLTA  Acquisition  Program  Baseline  transition. 

The  May  27,  2005,  approved  Acquisition  Program  Baseline  is  a  result  of  a  breach 
in  the  AHLTA  program’s  schedule.  The  breach  in  the  schedule  was  a  direct  result 
of  performance  problems  during  the  full  deployment  of  Block  1. 


AHLTA- Approved  Acquisition  Program  Baselines 

Acquisition  Program 
Baseline 

Acquisition  Program 
Baseline  Change  1 

Baselined  Acquisition 
Program  Baseline 

January  27,  2003 

November  17,  2003 

May  27,  2005 

Objective 

Threshold 

Objective  1  Threshold 

Objective  1  Threshold 

No  Changes  Unless 
Specified 

No  Changes  Unless 
Specified 

Milestone  (MS)  0 

JAN  1997 

APR  1997 

MS  1 

MAY  1998 

AUG  1998 

Block  1  Developmental 

Test  and  Evaluation 

JUN  2000 

SEP  2000 

Block  1  Operational  Test 
and  Evaluation 

APR  2002 

JUL  2002 

Block  1  MS  C  Limited 
Deployment 

NOV  2002 

MAY  2003 

Block  1  Full  Rate 
Production  Decision 

JUL  2003 

JAN  2004 

Block  2  System 
Requirements  Review 

NOV  2000 

FEB  2001 

Block  2  MS  B 

NOV  2002 

MAY  2003 

Block  2  Operational  Test 
Readiness  Review  3 

APR  2003 

OCT  2003 

NOV  2003 

MAY  2004 

DELETED 

Block  2  Release  1 
Deployment  Decision 
Review  (DDR) 

MAR  2005 

SEP  2005 

Block  2  MS  C 

JUL  2003 

JAN  2004 

MAR  2004 

SEP  2004 

DELETED  | 

Block  2  Full  DDR 

JUN  2006 

DEC  2006 

Block  3  MS  A 

NOV  2002 

MAY  2003 

Block  3  MS  B 

JAN  2004 

JUL  2004 

MAR  2006 

SEP  2006 

Block  3  MS  C 

JAN  2004 

JUL  2004 

SEP  2007 

MAR  2008 

Block  3  Release  1  DDR 

MAR  2008 

SEP  2008 

Block  3  Release  2  DDR 

MAR  2008 

SEP  2008 

Block  3  Release  3  DDR 

DEC  2008 

JUN  2009 

Block  3  Full  DDR 

SEP  2009 

MAR  2010 

Initial  Operating 

Capability 

MAR  2004 

SEP  2004 

Full  Operating  Capability 

SEP  2007 

SEP  2008 

SEP  2011 

SEP  2012 

Block  4  MS  A 

JUL  2003 

JAN  2004 

DELETED 

Block  4  MS  B 

SEP  2004 

MAR  2005 

DELETED 

Block  4  MS  C 

SEP  2004 

MAR  2005 

DELETED 
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Appendix  D.  Armed  Forces  Health  Longitudinal  Technology  Application 

Risk  Management  Reports 


Active  Risk  Management  Report 

Ai  of  Monday.  October  03.  3005  4:35:32  PM 

Risk-Id:  2005-020  Risk  Level:  Project  Key  Group:  Integration  Transfer  Office: 


Office  Of  Primary  Responsibility 

Start 

Date 

Close 

Date  Project  Name 

Priority  Risk  Source 

Impact  Area 

Gov't  POC  Project 
Officer 

CHCS  II  Project  Office 

8.1-2005 

cucsn 

3  Funding 

Cost 

CAPT  Heidi  Moos 

Risk  C  bamp :  Tracey  Brown 

Add'l  Projects)  Affected  by  Risk; 

Risk  Statement:  There  is  a  potential  consent  thai  the  coaaplexity  of  the  COTS  xTegration  may  result  in  costs  being  unde-stated 
Impact:  This  could  cause  program  lifecycle  costs  to  be  understated 


Mitigation  Strategy# 

Description 

Start 

Closed 

MitSME  Alt  \Dt_SME 

Summary: 

Date 

Date 

2005-020-1 

PO  is  currently  evaluating  mitigation  strategies. 

9282005 

Col  Tom  Beach  Traces-  Brown 

Mitigation  Strategy# 

Description 

Start 

Closed 

MitSME  Alt  Mit_SME 

Summarv: 

Date 

Date 

2005-020-2 


PO  is  currently  evaluating  mitigation  strategies. 


9  282005 


Col  Tom  Beach  Tracev  Brown 


K> 

O 


Appendix  D.  Armed  Forces  Health  Longitudinal  Technology  Application 

Risk  Management  Reports  (cont’d) 

Risk  Management  Report 

As  of  Thundqy  .  March  02.  2006 1 1: 02: 36 AM 

Risk-Id:  2005-020  P  Risk  Level:  Project  Key  Group:  Integration  Transfer  Office: 


Office  Of  Primary  Responsibility 

Start 

Date 

Close 

Date  Project  Name 

Priority  Risk  Source 

Impact  Area 

Gov't  POC  Project 
Officer 

CHCS  n  Project  Office 

812005 

CHCSH 

I  Funding 

Cost 

CAPT  Heidi  Moos 

Risk  C  hamp :  Tracey  3rewn 

Add'l  Projects)  Affected  by  Risk: 

Risk  Statement:  .aexe  is  the  potential  that  current  cost  assumptions  may  not  have  accounted  for  recently  acknowledged  discovered  cotuple vines  of  COTS  integration  resulting 
in  understated  cost  estimates  in  the  POM  sears 

Impact:  This  could  cause  schedule  delays,  cost  overruns,  and  performance  unpacts 

General  117>'200(S  The  CHCS  II  Project  Officer  and  the  CHCS  E  Engineering  Team  advised  the  PM  of  cost  and  complexity  concerns  with  the  COTs 
Comment;:  Integration,  and  based  on  tbeit  discussion  this  risk  has  been  elevated  by  the  CI7P0  PM  from  ?2  to  PI. 

ITT  2005  During  November's  CHCS  II IPR.  the  CHCS  II  Project  Officer  elevated  this  ask  fiom  a  priority  3  to  a  priority  2  based  on  die 
complexity  of  integrating  die  COTs  Prodncts  m  Block  3. 

9, 28  2005  Received  approval  fiotn  PO  to  open  two  mitigation  strategies  identified  by  CHCS  II  Project  Team. 

85  2005  PO  approved  nsk  as  Priority  3  Mils  are  being  coordinated  by  PO  and  Project  team. 

8.1  '2005  Hus  potential  fi.sk  was  identified  Awaiting  Project  Officer  approval 


Mitigation  Strategy* 
Summary: 

Description 

Start 

Date 

Closed 

Date 

MitSME  Alt  Mit_SME 

2005-020-1 

Coordinate  across  IM/TT  teams  during  FYOS  POM 
development  cycle. 

9282005 

Beach  Tom  Col  Brown  Tracey 

Mitigation  Strategy  # 
Summary: 

Description  Start  Closed  MitSME  '  Alt  Mit_SME 

Date 

Date 

2005-020-2 

Address  in  POM  submissions 

9282005 

Beach  Tom  Col  Brown  Tracey 

Appendix  D.  Armed  Forces  Health  Longitudinal  Technology  Application 

Risk  Management  Reports  (cont’d) 

Risk  Management  Report 

As  of  Friday  Februar,  24,  200b  1  :il  :4b  PM 

Risk-Id:  2004-030  P  Risk  Level:  Project  Key  Group:  Interoperability  Transfer  Office: 

Start  Close 

Office  Of  Primary  Responsibility  Date  Date  Project  Name  Priority  Risk  Source 

CHCSH  Project  Office  112005  CHCS  H  Block  3  2  Supporability 

Risk  Champ:  Tracey  Brown  Add  I  Projects)  Affected  by  Risk- 

Risk  Statement:  Legacy  CHCS  is  in  ±e  process  of  deploying  full  lab  interoperability.  Thus  allowing  data  sharing  among  DoD  labs  at  a  regional  level  and  with,  commercial 
reference  labs,  once  LAB  AP  COTS  is  deployed  thus  tbs  legacy  interoperability  will  be  lost  as  legacy  CHCS  lab  modules  are  tinned  off. 

Impact:  This  may  cause  significant  impact  inside  laboratory',  and  could  cause  moderate  impact  at  the  commander  level 

General  114  2 00?  Tne  original  start  date  6  S  2004.  Based  upon  ±e  changes  m  Block  3  4.  date  moved  out  to  be  reassessed  by  CITPO  PO  whether  or 
C  omments :  not  this  nsk  should  continue  to  be  tracked.  [DC] 

Mitigation  Strategy#  Description  Start  Closed  MitSME  ' Alt Mit_SME 

Summary:  Date  Date 

2004-OSO-l  P  Design  a  method  to  deploy  Lab  AP  COTs  by  region.  [Most  1  12005  Granado  Joseph  LCDR 

of  the  interoperability  happens  at  the  regional  level. 

Replacing  legacy  by  region  will  greatly  reduce  the 
disruption  to  legacy  interoperability.) 


Gov't  POC  Project 
Impact  Area  Officer 

Performance  LCDR  Joseph  Granado 


Appendix  D.  Armed  Forces  Health  Longitudinal  Technology  Application 

Risk  Management  Reports  (cont’d) 


Risk  Management  Report 

As  of  Friday  February  24.  2006  1 : 32:51  PM 


Risk-Id:  2004-03?  Rule  Level:  Project  Key  Group:  Convergence 


Transfer  Office: 


Office  Of  Primary  Responsibility 

CHCS  H  Project  Office 
Risk  C  bamp:  Tracey  3rown 


Start  Close 

Date  Date 


1  1  2005 


Project  Name 


chcse 


Add'l  Projects)  Affected  by  Risk: 


Priority  Risk  Source  Impact  Area 


Gov't  POC  Project 
Officer 


2  Supporability  Perfmmmce  LCDR  Joseph  Granadjo 


Risk  Statement:  Undefined  internal  interfaces  could  cause  some  integration  impacts  between  CHCS  II  and  the  C'OTs  product 
Impact:  Once  die  new  COTs  product  is  in  place  there  may  be  integrtioa  impacts  to  CHCS  IL 

General  114 '200?  The  original  start  date  6  S  2004.  Based  upon  ±e  changes  x  3 loci*.  3/4,  date  moved  our  to  be  reassessed  by  CITPO  PO  whether  on 
C ouunents :  not  this  risk  should  continue  to  be  tracked  [DC] 


Mitigation  Strategy  #  Description 
Summary; 


Closed  MitS.ME  Alt  Mit  SME 
Date 


2004-OS?-1  Continue  to  interface  developed  conflicts  with  the  new 
production  CHCS  H. 


1  12005 


Granado  Joseph  LCDR 


Appendix  D.  Armed  Forces  Health  Longitudinal  Technology  Application 

Risk  Management  Reports  (cont’d) 


Risk  Management  Report 

As  of  FridiQ.Februar,  24.  2006  2:23:22  PM 


Risk-Id:  2004-036  Risk  Level:  Project  Key  Group:  Integration 


Transfer  Office: 


Office  Of  Prunnrv  Responsibility 

CHCSn  Project  Office 
Risk  Champ:  Tracey  3rown 


Date  Date  Project  Name 

6  10:2004  CHCS  H 

Add’l  Projects)  Affected  by  Risk: 


Priority  Risk  Source 
2  Technical 


Impact  Area 
Performance 


Gov't  POC  Project 
Officer 

CAPT  Heidi  Moos 


Risk  Statement:  There  is  a  possibility  that  COTS  version  upgrades  may  not  be  included  m  the  CHCS II  baseline  when  available,  causing  unknown  technical  performance  issues. 

Impact:  If  COTS  product  upgrades  are  not  added  to  the  CHCS  II  baseline  m  a  timely  manner,  product  suppcrtabikry  and  fiincronality.  as  well  as  security  vulnerability 
fixes.  would  be  degraded  and  not  maintained  Interoperability  with  other  COTS  products  would  also  be  impacted 

General  9  3  2004  This  risk  w’as  downgraded  and  approved  to  a  priority  2  in  the  PR  held  24  August  2004.  (RM9304) 

Comments:  6  29  2  004  Risk  Background 

There  are  a  number  of  C  ommercial  Off-the  Shelf  (COTS)  products  that  comprise  the  CHCS  n  system,  including  Oracle  db.  Tuxedo.  3M, 
and  the  hardware  operating  system.  As  the  COTS  vendors  upgrade  product  versions  and  or  produce  software  patches  to  fix  bugs  or 
securin'  vulnerabilities,  these  upgraded  versions  must  be  added  to  the  CHCS  II  system  in  a  manner  that  does  not  degrade  current 
functionality  Planning  for  die  interoperability  ft  regression  testing,  as  well  as  subsequent  release  of  the  COTS  product  upgrades, 
remains  a  challenge  both  in  terms  of  cost  and  schedule.  [CAPT  Moos.  TB.  AK,  6  29  04] 

610  2004  New  proposed  risk  added  per  the  CHCS  II  PM-Level  Monthly  IPEL  [CAPT  Moos.  TB.  AK.  6  10  04] 


Mitigation  Strategy# 
Summarv: 


Description 


Closed 

Date 


MitSME  Alt  Mil  SME 


2004-086-1  Identify  conflicts  in  software  COTS  upgrades  and 

convergence  operational  issues  early  in  the  planning  and 
development  cycle. 


6  29  2004 


Moos  Keidt  CAPT  Brown  Tracev 
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5  122005  No  change  in  status  -  Recommended  contract  language  being  finalized. [Col  Beach.  TB.  DC.  S  12  05] 

7  14 2005  Recommended  contiact  language  being  5nalized[Col  Beach.  TB.  DC,  7  1405] 

6  10  2005  Recommendation:  for  managing  C  OTS  upgrades  were  briefed  to  CTTPO  PM  in  April  05  in  preparation  for  the  HPT  and  OIPT 

briefings.  Activity  continues  [Col  Beach,  TB.  DC,  d'1005] 

5  42005  Recommendations  foe  managing  COTS  upgrades  were  briefed  to  CTTPO  PM  in  April  05  mpreparation  for  the  HPT  and  OIPT 

briefings.  Activity  continues  [T  Brown.  DCC  4  May  05] 

4T42005  No  Update 

3/142005  No  Update 

2/15/2005  No  Update 

1  14 2005  CTTPO  IA  provided  updated  process  for  updating  system  hardware  and  data  base  with  security  patches  as  they  are  identified 

Draft  process  was  provided  to  Lrtegic,  DISA  CHCS II  teams.  CTTPO  CM,  Release  Management  and  TIMPO  for  review  and 
comment  C  omments  due  back  1  20  05.  A  follow-on  meeting  will  be  held  to  finalize  the  process.This  captures  one  aspect  of 
COTS  upgrades  CITPO  SE  developing  options  for  consideration  for  managing  C  OTS  upgrades.  [T  Brown,  13  Jan  05] 

122/2004  CITPO  Information  Assurance.  DISA,  Integic  (software  integrator).  Project  Team  and  CITPO  CM  continuing  to  woih  to  to  refine 
and  communicate  the  process  for  updating  system  hardware  and  data  base  with  updated  security  patches  New  Systems 
Engineering  contiact  with  Integic  for  2005  contains  a  task  that  requires  Integic  to  register  the  CHCS  II  system  components  on 
the  VMS  to  enable  them  to  receive  the  latest  security  notifications.  [T.  Brown,  2  Dec  04] 


1 1  16  200  CITPO  Infoimation  Assurance,  DISA  Integic  (software  integrator).  Project  Team  and  CTTPO  CM  working  together  to  refine  and 
communicate  the  process  for  updating  system  hardware  and  data  base  with  updated  security  patches  Basic  Process:  software 
security  packages  provided  to  CITPO  CM  by  DISA  passed  to  Integic  for  testing  and  integration  updating  of  documentation, 
passed  to  DTE  (via  CITPO  CM)  for  independent  testing  and  released  via  standard  CHCS  II  release  management  procedures 
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Risk  Management  Reports  (cont’d) 

W13  200  CITPO  SE  finalizing  white  paper  identifying  options  fox'  addressing  this  issue.  Integic  contracts  being  evaluated  to  determine 
what  tasks  are  covered  by  current  contracting  vehicles  (T  Brown  10/15  04) 

9  16  2004  QTPO  SE  is  drafting  white  paper  identifying  optioars  for addressing  this  issue  Draft  paper  due  September  04  (T  Broun 
9/15.04) 


6  29  2004  The  CHCS  II  developer  (Integic)  is  establishing  an  internal  risk  management  team  to  include  PGUI.  CHCS  II-T.  and  CKCS II 
software  developers  [C  APT  Moos,  TB.  AK.  6/29  04] 

to  - 

ryi 
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Office  of  the  Assistant  Secretary  of  Defense 
(Health  Affairs)  Comments 


THE  ASSISTANT  SECRETARY  OF  DEFENSE 

1  200  DEFENSE  PENTAGON 
WASHINGTON,  DC  2030 1-1 200 

HEALTH  AFFAIRS 

APR  7  pmr 

Mr.  Richard  Jolliffe  ^ 

Office  of  the  Inspector  General 

Department  of  Defense 

400  Army  Navy  Drive,  Room  300 

Arlington,  VA  22202-4704 

Dear  Mr.  Jolliffe: 

This  is  the  Office  of  the  Assistant  Secretary  of  Defense  (Health  Affairs) 
/TRICARE  Management  Activity  (TMA)  response  to  the  recommendations  in  the 
Department  of  Defense  (DoD)  Inspector  General  (IG)  draft  audit  report,  “Audit  of  the 
Acquisition  of  the  Armed  Forces  Health  Longitudinal  Technology  Application,” 

January  25,  2005  (Project  No.  D20O5-DO0AS-0 11 7.0000). 

The  TMA  acknowledges  receipt  of  the  proposed  draft  audit  report  and  concurs  with 
the  overall  recommendations.  Specifically,  TMA  will  develop  additional,  more  robust 
mitigation  strategies  associated  with  Commercial  Off-The-Shelf  integration  risk. 
However,  TMA  takes  exception  to  several  inaccurate  statements,  to  include  “the  program 
office  has  not  identified  any  mitigation  strategies.”  TMA  provided  corrections  to  these 
statements  in  the  March  3  Feedback  to  the  Discussion  Draft  Report,  to  include  four 
Armed  Forces  Health  Longitudinal  Technology  Application  risk  management  reports 
with  corresponding  mitigation  strategies.  Only  one  of  these  was  utilized  in  the  report  and 
included  in  Appendix  D.  The  omission  of  these  key  documents  misrepresents  the 
proactive  risk  management  efforts  taken  within  the  program  office. 

Enclosed  are  specific  TMA  responses  to  the  DoD  IG’s  draft  audit  report.  Please 
feel  free  to  direct  questions  on  this  matter  to  my  project  officer,  Ms.  Pamela  Schmidt,  at 
(703)  681-8830,  or  Mr.  Gunther  Zimmerman  (General  Accounting  Office  IG  Liaison),  at 
(703)  681-3492. 

Sincerely, 


Enclosures: 
As  stated 


Inspector  General  (IG)  Report-Dated  March  24,  2006 

Inspector  General  (IG)  Draft  Audit  Report 
“Acquisition  of  the  Armed  Forces  Health  Longitudinal  Technology  Application” 
Project  No.  D2005-DO00AS-01 17.000 
Office  of  the  Secretary  of  Defense  (Health  Affairs)  Response 

RECOMMENDATION  1:  We  recommend  that  the  Program  Manager,  Armed  Forces 
Health  Longitudinal  Technology  Application,  provide  documentation  that  supports  the 
program  management  office  decisions  on  November  1,  2005,  and  January  17,  2006,  that 
increased  the  risk  priority  value  for  commercial  off-the-shelf  product  integration  into  the 
Armed  Forces  Health  Longitudinal  Technology  Application  from  Priority  Value  3 
(medium)  to  Priority  Value  2  (high),  and  from  Priority  Value  2  (high)  to  Priority  Value  1 
(high). 

OASD  (HA)  RESPONSE:  Concur.  The  Program  Manager  will  provide  the  appropriate 
documentation  to  support  the  assignment  of  risk  priorities  associated  with  commercial 
off-the-shelf  product  integration. 

RECOMMENDATION  2:  We  recommend  that  the  Program  Manager,  Armed  Forces 
Health  Longitudinal  Technology  Application,  provide  justification  and  an 
implementation  plan  for  the  Priority  Value  1  (high)  risk  assigned  to  Block  3. 

OASD  (HA)  RESPONSE:  Concur.  The  Program  Manager  will  provide  justification 
and  an  implementation  plan  for  the  Priority  Value  1  (high)  risk  assigned  to  AHLTA 
Block  3. 

RECOMMENDATION  3:  We  recommend  that  the  Program  Manager.  Armed  Forces 
Health  Longitudinal  Technology  Application,  develop  additional  or  more  robust 
mitigation  strategies  that  address  the  commercial  off-the-shelf  product  integration 
Priority  Value  1  (high)  risk  in  accordance  with  the  CITPO  Risk  Management  Plan.  These 
mitigation  strategies  should,  at  a  minimum,  contain  the  recommendations  included  in  the 
Risk  Management  Guide  for  DoD  Acquisition,  Fifth  Edition,  version  2.0,  June  2003  [to] 
develop  additional,  more  robust  mitigation  strategies  to  further  reduce  and  control  this 
risk. 

OASD  (HA)  RESPONSE:  Concur.  The  Program  Manager  will  continue  to  develop 
additional  and  more  robust  mitigation  strategies  that  address  the  commercial  off-the-shelf 
product  integration  Priority  Value  1  (high)  risk  in  accordance  with  the  CITPO  Risk 
Management  Plan.  Corresponding  mitigation  strategies  will  adhere  to  the 
recommendations  included  in  the  Risk  Management  Guide  for  DoD  Acquisition,  Fifth 
Edition,  version  2.0,  June  2003. 
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Inspector  General  (IG)  Draft  Audit  Report 
“Acquisition  of  the  Armed  Forces  Health  Longitudinal  Technology  Application” 
Project  No.  D20O5-D00OAS-01 17.000 
Office  of  the  Secretary  of  Defense  (Health  Affairs)  Response 

TMA  TECHNICAL  COMMENTS: 

Executive  Summary 

Current:  (Paragraph  3) 

“Additionally,  the  program  management  office  has  not  identified  any  mitigation  strategies  to 
reduce  and  control  risk”. 

TMA  Resuonse: 

The  program  management  office  did  identify  several  mitigation  strategies.  This  issue  was 
identified  in  the  TMA’s  3  Mar  2006  feedback  to  the  Discussion  Draft  of  a  Proposed  Report, 
which  included  the  following  Risk  Management  reports.  Additionally,  only  one  of  these  reports. 
2005-020.  is  listed  in  Appendix  D.  Recommend  all  four  reports  be  included. 

2005-020,  CHCSII  COTS  Integration.  1  Aug  2005 
2004-080,  CHCSII.  Block  III  -Lab  AP/COTS  Interoperability,  1  Jan  2005 
2004-085.  CHCSII,  COTS  Integration/Convergence,  1  Jan  2005 
2004-086,  CHCSII,  COTS  Integration,  10  Jun  2004 

Classification  of  Commercial  Off-the-Shelf  Risk 

Current:  (Page  5.  paragraph  1) 

"Additionally,  the  program  management  office  has  not  identified  any  mitigation  strategies  to 
reduce  and  control  program  risk.” 


I  MA  Response: 

The  program  management  office  did  identify  mitigation  strategies.  This  issue  was  identified  in 
the  TMA’s  3  Mar  2006  feedback  to  the  Discussion  Draft  of  a  Proposed  Report,  which  included 
the  following  Risk  Management  reports.  Additionally,  only  one  of  these  reports.  2005-020,  is 
listed  in  Appendix  D.  Recommend  all  four  reports  be  included. 

2005-020.  CHCSII  COTS  Integration.  1  Aug  2005 
2004-080,  CHCSII.  Block  III  -Lab  AP/COTS  Interoperability,  1  Jan  2005 
2004-085,  CHCSII,  COTS  lntegration/Convergenee.  1  Jan  2005 
2004-086,  CHCSII,  COTS  Integration,  10  Jun  2004 


31 


Final  Report 
Reference 


Page  9, 
paragraph  3 


Page  9, 
paragraph  4 


Mitigation  Strategies 

Current:  (Page  8,  paragraph  6) 

“At  the  time  of  our  review,  the  Program  Management  Office  did  not  have  a  mitigation  strategy 
associated  with  the  identified  COTS”. 


TMA  Response: 

The  program  management  office  did  identify  mitigation  strategies.  This  issue  was  identified  in 
the  TMA’s  3  Mar  2006  feedback  to  the  Discussion  Draft  of  a  Proposed  Report,  which  included 
the  following  Risk  Management  reports.  Additionally,  only  one  of  these  reports,  2005-020,  is 
listed  in  Appendix  D.  Recommend  all  four  reports  be  included. 

2005-020,  CHCSII  COTS  Integration,  1  Aug  2005 
2004-080.  CHCSII,  Block  HI  -Lab  AP/COTS  Interoperability,  1  Jan  2005 
2004-085,  CHCSII,  COTS  Integration/Convergence,  1  Jan  2005 
2004-086,  CHCSII,  COTS  Integration,  10  Jun  2004 


AHLTA  Program  Management  Office  Response  to  Discussion  Draft 

Current:  (Page  9.  Paragraph  2) 

“In  response  to  our  request  for  additional  information  to  support  their  statement,  the  program 
office  staff  provided  an  updated  copy  of  their  COTS  Integration  Risk  Management  Report, 

March  2, 2006,  which  showed  that  on  September  28,  2005,  the  project  officer  approved  opening 
two  mitigation  strategies  identified  by  the  CHCS  II  Project  Team.  However,  this  current  report 
contradicts  the  risk  report  provided  to  the  audit  team  on  October  3,  2005,  because,  according  to 
the  risk  report  at  that  time,  the  program  office  was  still  evaluating  mitigation  strategies. 

Appendix  D  contains  the  October  3,  2005,  and  March  2, 2006,  Risk  Management  Reports." 

TMA  Response:  TMA  does  not  see  a  contradiction  in  these  reports- perhaps  some  explanation  is 
needed.  The  Risk  Management  report  (2005-020)  documents  the  status  of  the  COTS  integration 
risk  at  that  time.  As  of  3  Oct  2005,  mitigation  strategies  2005-20-1  and  2005-20-2  indicate  the 
Program  Office  was  evaluating  mitigation  strategies,  which  were  initiated  28  Sep  2005.  The  2 
Mar  06  risk  management  report  2005-20  provides  traceability  of  updates  to  this  risk  management 
report,  reflected  in  the  General  Comments  section: 


1  Aug  2005: 
5  Aug  2005: 
28  Sep  2005: 
1  Nov  2005: 
17  Jan  2006: 


Potential  risk  identified,  awaiting  approval 

Priority  level  3  risk  assigned 

Approval  to  develop  2  mitigations  strategies 

Approval  to  elevate  risk  from  level  3  to  level  2 

PM  is  advised  of  cost  and  complexity  of  COTS  integration; 

risk  elevated  to  level  1 
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In  addition  to  Risk  Management  Report  2005-020,  die  program  office  provided  the  following 
risk  management  reports  3  Mar  2006,  which  were  not  included  in  Appendix  D: 

2004-080,  CHCS1I,  Block  111  -Lab  AP/COTS  Interoperability,  1  Jan  2005 
2004-085,  CHCSII,  COTS  Integration/Convergence,  1  Jan  2005 
2004-086,  CHCSII,  COTS  Integration,  10  Jun  2004 

Conflicting  Priority  Values  of  the  Risk  (Page  10,  Paragraph  1) 

TMA  Response: 

The  statement,  “Conflicting  Priority  Values  of  the  Risk"  is  misleading  and  misrepresents  the 
CITPO  Risk  Management  Program.  These  changes  reflect  proactive  adjustments  made  to  the 
AHLTA  COTS  integration  risk  2005-020  priorities  over  time,  from  5  Aug  2005  (Level  3),  1  Nov 
2005  (Level  2),  and  17  Jan  2006  (Level  1)  Suggest  “Conflicting”  be  changed  to  “Adjusting  ”, 

Current:  (Page  10,  Paragraph  1) 

“The  program  management  office  comments  to  the  discussion  draft  report  also  identified  COTS 
integration  as  a  medium  risk.” 

TMA  Response: 

This  sentence  has  apparently  been  misinterpreted  and  taken  out  of  context.  In  the 
3  Mar  2006  comments  to  the  discussion  draft  report,  TMA  provided  the  following  statement, 
“CITPO  has  identified  COTS  integration  as  a  medium  level  program  risk  and  developed 
corresponding  mitigation  strategies.”  The  statement  was  intended  to  refute  the  following,  “the 
program  office  has  not  identified  any  mitigation  strategies”. 

This  inaccurate  statement  is  repeated  in  the  Executive  Summary  and  on  pages  5  and  8,  even 
though  TMA  provided  corrections  3  Mar  2006.  CITPO  provided  four  COTS  risk  management 
reports  dating  back  to  10  Jun  2004,  with  risk  priority  assignments  that  range  from  1  to  3.  These 
risk  priorities  have  been  adjusted  over  time  to  help  manage  AHLTA’s  COTS  integration  risk. 


Page  10, 
paragraph  2 


Page  10, 
paragraph  2 
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